Skip to content

feat(driver-kubernetes): add empty_dir workspace volume with bounded size - #4473

Open
kon-angelo wants to merge 1 commit into
NVIDIA:mainfrom
kon-angelo:feat/2787-workspace-volume-empty-dir/kon-angelo
Open

feat(driver-kubernetes): add empty_dir workspace volume with bounded size#4473
kon-angelo wants to merge 1 commit into
NVIDIA:mainfrom
kon-angelo:feat/2787-workspace-volume-empty-dir/kon-angelo

Conversation

@kon-angelo

@kon-angelo kon-angelo commented Oct 13, 2026

Copy link
Copy Markdown

Summary

Add a workspace_volume option to the Kubernetes compute driver so a gateway can back each sandbox's /sandbox with a size-capped emptyDir instead of a per-sandbox PVC. This lets sandboxes run on clusters that have no dynamic storage provisioning or no default StorageClass, and takes the PVC lifecycle (orphaned claims, reclaim policy) off the teardown path — kubelet reclaims the emptyDir with the Pod. Default is unchanged (persistent_volume_claim).

Related Issue

Closes #2787

Changes

  • workspace_volume gateway setting (persistent_volume_claim default | empty_dir); default preserves current behaviour byte-for-byte.
  • empty_dir omits volumeClaimTemplates on the Sandbox CR and mounts a pod-level emptyDir at /sandbox, with sizeLimit sourced from workspace_default_storage_size. The workspace-init container still seeds /sandbox from the image on every pod start (the volume is empty each start).
  • workspace_max_storage_size (default 10Gi) bounds per-sandbox size overrides so one sandbox cannot claim an unbounded workspace and starve the node; startup validation rejects a default above the maximum.
  • Per-sandbox overrides via driver_config.kubernetes.workspace_volume and workspace_storage_size, honoured only on gateways that set allow_driver_config. A requested size above the gateway maximum is rejected; workspace_storage_class applies to PVCs only.
  • Standalone driver env: OPENSHELL_K8S_WORKSPACE_VOLUME, OPENSHELL_K8S_WORKSPACE_MAX_STORAGE_SIZE.
  • Small in-crate quantity parser (IEC/SI suffixes) for the size comparison; no new dependency.

Testing

  • Checks appropriate to the affected code and behavior pass (cargo test -p openshell-driver-kubernetes: 304 pass; cargo clippy clean; openshell-server builds)
  • Unit tests added/updated (serde, quantity parser, render diff for both modes, override precedence both directions, size→sizeLimit, size→PVC request, over-max rejection, unknown-value rejection)
  • E2E tests added/updated (if applicable)

Also verified manually on a local k3d cluster deployed with workspace_volume = empty_dir + allow_driver_config = true:

  • Sandbox reaches Ready with no workspace PVC; CR has no volumeClaimTemplates; Pod volume is emptyDir={"sizeLimit":"2Gi"}; workspace-init seeds /sandbox.
  • Per-sandbox workspace_storage_size: 5Gi → sizeLimit: 5Gi; 50Gi (> 10Gi max) → rejected with a clear error.
  • Marker written to /sandbox is gone after stop/start (ephemeral contract).

Checklist

  • Follows Conventional Commits
  • Commits are signed off (DCO)

Draft — remaining before ready for review: config docs (docs/how-it-works/gateways/configuration.mdx, docs/how-it-works/sandboxes/runtimes.mdx) are not yet updated for workspace_volume / workspace_max_storage_size; no Rust e2e lane added (an empty_dir gateway fails the default sandbox-lifecycle conformance by design).

@copy-pr-bot

copy-pr-bot Bot commented Oct 13, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@github-actions

github-actions Bot commented Oct 13, 2026 •
edited
Loading

Copy link
Copy Markdown

All contributors have signed the DCO ✍️ ✅
Posted by the DCO Assistant Lite bot.

@kon-angelo
kon-angelo marked this pull request as ready for review October 15, 2026 09:36
@dimityrmirchev dimityrmirchev mentioned this pull request Oct 15, 2026
Closed
5 of 6 tasks
@kon-angelo

kon-angelo commented Oct 15, 2026

Copy link
Copy Markdown
Author

I have read the DCO document and I hereby sign the DCO.

@kon-angelo
feat(driver-kubernetes): add empty_dir workspace volume with bounded …
2285718 
…size

Add a `workspace_volume` option to the Kubernetes compute driver so a
gateway can back each sandbox's /sandbox with a size-capped emptyDir
instead of a per-sandbox PVC. This lets sandboxes run on clusters that
have no dynamic storage provisioning or no default StorageClass, and
removes the PVC lifecycle (orphaned claims, reclaim policy) from the
teardown path — kubelet reclaims the emptyDir with the Pod.

- workspace_volume (persistent_volume_claim | empty_dir); default
  persistent_volume_claim, so existing behaviour is unchanged.
- empty_dir omits volumeClaimTemplates on the Sandbox CR and mounts a
  pod-level emptyDir at /sandbox, sized from workspace_default_storage_size
  as the sizeLimit. The workspace-init container still seeds /sandbox from
  the image on every pod start (the volume is empty each start).
- workspace_max_storage_size (default 10Gi) bounds per-sandbox size
  overrides so a single sandbox cannot claim an unbounded workspace and
  starve the node; startup validation rejects a default above the maximum.
- Per-sandbox overrides via driver_config.kubernetes.workspace_volume and
  workspace_storage_size, honoured only on gateways that set
  allow_driver_config. A requested size above the gateway maximum is
  rejected; workspace_storage_class applies to PVCs only.
- Standalone driver env: OPENSHELL_K8S_WORKSPACE_VOLUME and
  OPENSHELL_K8S_WORKSPACE_MAX_STORAGE_SIZE.

Refs: NVIDIA#2787
Signed-off-by: Konstantinos Angelopoulos <konstantinos.angelopoulos@sap.com>
@kon-angelo
kon-angelo force-pushed the feat/2787-workspace-volume-empty-dir/kon-angelo branch from 499fd96 to 2285718 Compare October 21, 2026 07:02
@juancarlosm

juancarlosm commented Oct 30, 2026

Copy link
Copy Markdown

Another data point for #2787 / #4473, from a production EKS cluster (OpenShell 0.1.5, Kubernetes driver, workspace_mode = shared).

We run short-lived agent sandboxes whose workload never writes to /sandbox: its data lives elsewhere in the container and its durable state goes to S3 snapshots. The injected workspace PVC (2Gi, gp3 EBS, cluster default StorageClass) is the largest single cost of a sandbox cold start:

Phase (one new sandbox) Time
Sandbox CR created +1.3s
Scheduling 3s
Workspace PVC provision + attach + mount ~12s
Containers start 3s
Workload ready + first reply ~5.6s
Total to first reply 21.9s

The PVC is more than half of the cold start, and a new sandbox is created for every new conversation.

The driver_config route does not help on 0.1.5: driver-config volumes accept only persistent_volume_claim, and it requires allow_driver_config = true, which we keep off. A gateway-level workspace_volume = "empty_dir" as in #4473 covers our case exactly with no per-sandbox config. We'd like to see it merged and are happy to test a build.

@amfred amfred mentioned this pull request Nov 9, 2026
Open
2 of 9 tasks
@Varshith-Kali Varshith-Kali mentioned this pull request Nov 11, 2026
Closed
5 of 6 tasks
@jhjaggars jhjaggars mentioned this pull request Nov 12, 2026
Open
2 tasks

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

feat(k8s): support ephemeral emptyDir-backed sandbox workspaces as an alternative to the default PVC

2 participants

@kon-angelo @juancarlosm